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ABSTRACT 

Lessons learned in changing a large-scale testing 
program are outlined. The redesigned Scholastic Aptitude Test (SAT) 
will be administered in spring 1994. Major changes for the verbal SAT 
section include more emphasis on critical reading, longer and more 
accessible passages, measurement of vocabulary in context, and use of 
double passages with two viewpoints. Changes to the mathematical SAT 
section include adding a subset of questions whore students must 
produce the responses and permitting, but not requiring, use of a 
calculator. Changes are also proposed for the SAT subject tests, 
including a writing tost at ail test administrations. The process of 
change will involve challenges, different perspectives, changes in 
conditions throughout the process, many viewpoints, hard-and-fast 
positions of some participants, resistance to change in general, 
unanticipated difficulties, and opposition to the finished product. 
Five lessons drawn from the process are: (1) tell the truth? (2) ask 
for help? (3) do not wait for permission? (4) keep in touch? and (5) 
be nice to each other. Thirteen overhead transparency masters 
illustrate points made in the presentation. (SLD) 
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^ • Thank you for coming to our session. What we will be sharing with you 

i> today is what we have learned as we have participated in a change process 

•mmi 

' for large scale testing programs. 

CO 
Q 

• My presentation has three parts: 

highlights of the test and the changes made (SAT) 

some things you can anticipate when changing a highly visible testing 
program (About 2 million students take the SAT each year.) 
some bits of advice — five lessons that I feel I have learned 



B. NATURE OF THE CHANGES 

• I have a handout ~ "A completely redesigned SAT will be administered in 
the Spring of 1994." 

• The inside cover lists highlights; that's what I will describe as a context 
for my comments. 
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• The College Board le using the name SAT-I to refer to the Verbal & 
Mathematical sections* 

Verbal section (major changes) 

* more emphasis on critical reading 

* longer, more accessible passages 

* measurement of vocabulary-ia-context (dropping antonyms) 

* use of double passage, two points of view 
Mathematical section {major changes) 

* adding a subset of questions where students must produce the 
response (no choices given/ 

* permitting, but not requiring the use of the calculator 

• Then in SAT-II: Subject Tests 

offering an SAT-Writing Test with a writing exercise at all subject 

test administrations 

new tests in Asian languages 

new English as a Second Language Proficiency Test 

t Changes have been studied with field trials at high schools and colleges — 
over 100,000 students, 

• Changes have been approved by the Trustees of the College Board. The new 
test will be introduced in the 1993-94 academic year. 

• The PSAT/NMSQT will first be introduced in the fall of 1993. 

• SAT changes will first be introduced in the spring of 1994. 
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If you have children entering the 9th grade this fall, and they take the 
PSAT/NMSQT at the start of their junior year, they would take the new test. 

I have been working on this project for four years . It has been the 
biggest event in my career, so I would be happy to talk in detail about the 
changes outside of this session with anyone interested. 

I want to move now, though, from the changes themselves to the process of 
change and what I can share from that experience that may be useful and 
perhaps a bit of fun. 

C. THINGS YOU CAN EXPECT 
Expect Challenges (See Transparency #1) 
Expect challenges. This is not work for 

low-energy people 

easily discouraged people 
The larger the program and the more people who care about it, the greater 
the number of challenges likely. 

plfferent Perspectives (See Transparency #2) 

Watch out for the fact that different people will bring different 
perspectives to the process. In the case of the New SAT, I found it meant 
different things to different people. 

A dinosaur -It will never change. 

An owl -It is a measure of wisdom. 

A babv - It is a baby compared to what it could be if it ever grew up to 
be a real assessment program. 



A skunk -As far as some teat takers on Saturday mornings are concerned, 
they think it stinks. 

Things Change (See Transparency #3) 
As you work, the " qivens " change: 

"Whatever you do, be sure you don't change the timing." 

"The answer sheet has to have these characteristics. " 

"We need longer or shorter score reports." 
When everything goes well, you get this kind of change. 

Many Points of View (See Transparency #4) 

When you bring specific ideas on prototypes to people to review, you find 
"many points of view." You need to understand why people have these 
reactions. What are their needs and how can you meet them? A lot depends 
on how you pose questions: 

Question : Do you think we should change the SAT? - Answer : N01 
Ask these questions first: 

Alternative Questions : How do you like these ideas? Do you think we 

should incorporate them into the SAT? Answers : I like them. Please 

make the changes. 

We Love Antonym s (See Transparency #5) 

People will take positions on issues you might not think they would even 
care about — like this fellow who rejects the idea that inferring 
vocabulary from the context is a good way to test. He wants antonyms 
(opposites) test items retained. Maybe he runs a coaching skill that 
drills on word lists. 



E xpect Rob i stance (See Transparency #6) 

Changing is unsettling to some people. You can expect resistance from 
within your family and outside. 

Harder Than You Think (See Transparency #7) 

The work will be harder than you expected. It will take longer, cost more, 
and make more demands than you expect. 

Good Work, I Hate It (See Transparency #8) 

When you are done, even if you were very careful about listening to users 
and made good changes, know that not everyone will love your work. 

We have a very large majority of people saying "Good workl We like the 
development process and outcome and can see how the changes pave the way 
for other good changes in the future." 

Others, a small group I am pleased to say, tell us, "I hate itl" 

D. FIVE LESSONS 
Tell the Truth (See Transparency #9) 

For this first one T will put up the symbol and urge you to call out an 
answer. (It's George Washington.) 
This symbol could mean 

- "Have the president's support . H 

or 

- "It's hard to be the president. H 



What I do mean to say is 

- "Tell the truth. " 
You have to work very hard to keep everyone informed even if there's a 
problem. Include a possible solution when you bring a problem to 
management. I spent close to half my own time on this. 

Also, don't sugar coat everything. Give straight stories. 

Ask for Help (See Transparency #10) 

My second lesson is Ask for Help l Call people, visit, hold meetings, and 
sometimes write memoranda (but we used these more to inform rather than to 
request help) • 

Admit you don't have all the answers or even perhaps that you don't know 
all the questions you should be asking. Practice being humble 1 

The New SAT project had the strong support of the President of ETS and of 
The College Board and of the other officers of both organizations. We got 
spectacular help from all staff. 

Don't Wait for Permission (See Transparency #11) 

This advice is particularly relevant for a large organization. The New SAT 
project had a very good set of procedures for administrative review. We 
had an approved work plan, schedule, and budget, so there really wasn't 
much risk that we would be very reckless. We were also building on solid 
past knowledge and trying to introduce evolutionary change. 



We still faced an enormous number of decision points regarding project 
design and direction. We tried to move forward within our plan and keep 
management informed rather than seek approval . 

Tell people as early as you can what you plan to do and what troubles you 
are encountering. Do not try to hide problems and think they will go away. 
If you move ahead but make a mistake, it's easy to ask forgiveness, but if 
you wait, you will still be waiting. 

Keep in Touch (See Transparency #12) 

Since the New SAT project was jointly funded by the College Board and 
ETf one clear place where we needed to keep in touch was among a fairly 
good sized set of people with lots of critical pairs such as the co-project 
directors at The College Board and ETS. 

Many such connections evolved within ETS: 
Test Developers and Psychometricians 
Test Developers and Test Production specialists 

Test Production specialists and Publications/Printing specialists 

Operations and Systems 
The written word is valuable but joint planning (face to face) is 
essential. 

Literally, "keeping in touch" is important. 
Be Nice to Each Other (See Transparency #13) 

Finally, and possibly most important, M Be nice to each other. " 
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Any big project is really a coordinated net of many projects of various 
sizes. A change process for a large scale testing program within a testing 
company can be r spared to the impact on a family moving to a new home, 
having a wedding, and planning a graduation party — all at the same ti&;2. 
Not all events will go as expected. Some will cause major troubles. When 
stakes are high and you feel pressure, it is easy to forget to take the 
other person's perspective and fail to treat people with the sensitivity 
they deserve. 

One of my goals for the New SAT project was for the College Board and ETS 
contributors to like and respect each other even more at the end of work 
than we did at the start. We are still working well together towards the 
introduction of our new tests, and we are also achieving well the goal of 
being nice to each other. This mutual respect and regard has already 
helped our project a great deal and continues to help us enormously. 
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Transparency #1 from presentation notes of 
John Fremer, Annual ECS/CDE Assessment 
Conference, Breckinridge, Colorado, June 1991 



(Original transparency is in color.) J I 



Perspectives ! 




Transparency //2 from presentation notes of 
John Fremer, Annaul ECS/CDE Assessment 
Conference, Breckinridge, Colorado, June 1991 

(Original transparency is in color)) 13 
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Transparency #3 from presentation notes of 
John Fremer, Annual ECS/CDE Assessment 
Conference, Breckinridge, Colorado, June 1991 



(Original transparency is in color.) i r 



Many Points 

of View ! 
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Transparency #4 from presentation notes of 
John Fremer, Annual ECS/CDE Assessment 
Conference, Breckinridge, Colorado, June 199] 



(Original transparency is in color «) 




Transparency #5 from presentation notes of 
(John Fremer, Annual EC3/CDE Assessment 
Conference, Breckinridge, Colorado, June 1991 

(Original transparency is in color.) 

Hi 




Expect 
Resistance ! 

Transparency #6 from presentation notes of 
John Fremer, Annual ECS/CDE Assessment 
Conference , Breckinridge , Colorado, June 1991 



(Original transparency is in color.) 
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transparency #7 from presentation notes of 
John Fremer. Annual ECS/CDE Assessment 
Conference, Breckinridge, Colorado, June 1991 

(Original transparency is in color.) 
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Good Work! 
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I Hate It! 

Transparency #8 from presentation notes of John Fremer, Annual 
ECS/CDE Assessment Conference, Breckinridge, Colorado, June 1991 
(Original transparency is in color.) 




Tell the Truth! 

•Transparency #9 from presentation notes of John Fcemer, Annual 
ECS/CDE Assessment Conference, Breckinridge, Colorado, June 1991 

(Original transparency is in color •) 
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Help! 



(Transparency #10 from presentation notes of 
John Fremer, Annual ECS/CDE Assessment 
Conference, Breckinridge, Colorado, June 199] 

(Original transparency is in color.) 
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Transparency #11 from presentation notes of 
Jbhn Fremer, Annual ECS/CDE Assessment 
Conference, Breckinridge, Colorado, June 1991 

(Original transparency is in color •) 
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Keep in Touch! 




Be Nice 




Transparency #13 froa presentation 
notes of John Freaer, Annual ECS/CD1 
Assessment Conference, Breckinridge, 
Colorado, June 1991 

(Original transparency is In color 



